{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SMFUgtvbFeLQ"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/docs/langchain-retrieval-augmentation.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/docs/langchain-retrieval-augmentation.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "v0to-QXCQjsm"
   },
   "source": [
    "# Retrieval-Augmented Generation with Pinecone, LangChain and OpenAI\n",
    "\n",
    "## Fixing LLMs that Hallucinate\n",
    "\n",
    "In this notebook, you'll learn one of the most common applications of retrieval-augmented generation: giving large language models access to up-to-date information. \n",
    "\n",
    "\n",
    "### Demo Data: Pinecone Documentation\n",
    "\n",
    "A great example to use RAG is when augmenting LLMs with information that may not exist in their training data. This could private data, internal company information, or data that has been updated post a training cutoff. In our case, many modern LLMs are rained on Pinecone data that has since been updated, such as release notes, quickstart guides, blog posts, docs, and code.\n",
    "\n",
    "In this example, we'll show the differences in generation from OpenAI's LLMs when asked about Pinecone's release notes! We'll orchestrate our RAG workflow using LangChain, a popular framework for AI applications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "VpMvHAYRQf9N",
    "outputId": "42705370-c5bf-417b-fe8a-3f6d0fe28fbb"
   },
   "outputs": [],
   "source": [
    "!pip install -qU \\\n",
    "    \"langchain[openai]\"\\\n",
    "    langchain-text-splitters==0.3.8 \\\n",
    "    langchain-pinecone==0.2.1 \\\n",
    "    pinecone-notebooks==0.1.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NhWnLkHqmeWI"
   },
   "source": [
    "---\n",
    "\n",
    "## Building a Knowledge Base with our Vector Database\n",
    "\n",
    "Building more reliable LLMs tools requires an external _\"Knowledge Base\"_, a database that we can query and update periodically with information.\n",
    "\n",
    "Specifically, we will need to retrieve information that is relevant to our queries. To do this we need to use _\"dense vector embeddings\"_. These can be thought of as numerical representations of the *meaning* behind our sentences.\n",
    "\n",
    "There are many options for creating these dense vectors, like open source [sentence transformers embedding models](https://www.pinecone.io/learn/series/nlp/) or OpenAI's [text-embedding-3-small model](https://platform.openai.com/docs/models/text-embedding-3-small). We will use OpenAI's offering in this example.\n",
    "\n",
    "### Before you begin...\n",
    "\n",
    "Be sure to grab a [free Pinecone account](https://app.pinecone.io/?sessionType=signup) and an OpenAI API key, [located here](https://platform.openai.com/api-keys)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Getting our Dataset: \n",
    "\n",
    "# These are markdown versions of our release notes from 2025 and 2024\n",
    "release_notes_2025 = \"https://docs.pinecone.io/release-notes/2025.md\"\n",
    "release_notes_2024 = \"https://docs.pinecone.io/release-notes/2024.md\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing our data\n",
    "\n",
    "We'll use Requests and LangChain to pull down the release notes, and process the associated Markdown. We've used splitters that correspond to the release year, month/year, and features. \n",
    "This will take us from raw Markdown files to LangChain Documents, which we'll embed and store in Pinecone."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "EI2iYxq16or9",
    "outputId": "55a8dd92-34e1-4237-a5c4-76016346999d"
   },
   "outputs": [],
   "source": [
    "# We'll grab these urls and parse them using LangChain's textsplitter for markdown\n",
    "from langchain_text_splitters import MarkdownHeaderTextSplitter\n",
    "from langchain_core.documents import Document\n",
    "import requests\n",
    "\n",
    "splitter = MarkdownHeaderTextSplitter(headers_to_split_on=[(\"#\", \"release\"), (\"##\", \"month_year\"), (\"###\", \"feature\")])\n",
    "\n",
    "def download_link(url):\n",
    "    response = requests.get(url)\n",
    "    response.raise_for_status()\n",
    "    return response.text\n",
    "\n",
    "\n",
    "def add_document_metadata(doc, new_metadata):\n",
    "    # returns new documents with updated metadata\n",
    "    old_metadata = doc.metadata\n",
    "    new_metadata = {**old_metadata, **new_metadata}\n",
    "    return Document(page_content=doc.page_content, metadata=new_metadata)\n",
    "\n",
    "\n",
    "def preprocess_pinecone_docs(urls):\n",
    "\n",
    "    pinecone_docs = []\n",
    "    for url in urls:\n",
    "        # download the markdown\n",
    "        response = download_link(url)\n",
    "        split_text = splitter.split_text(response)\n",
    "        # Update metadata to include url as source  \n",
    "        split_text = [add_document_metadata(doc, {\"source\": url, \"chunk_num\": num}) for num, doc in enumerate(split_text)]\n",
    "        pinecone_docs.extend(split_text)\n",
    "    return pinecone_docs\n",
    "\n",
    "\n",
    "pinecone_docs = preprocess_pinecone_docs([release_notes_2024,release_notes_2025])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a closer look at one of these notes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document content:  Pinecone Assistant can now [return a JSON response](/guides/assistant/chat-with-assistant#json-response).  \n",
      "***  \n",
      "You can now [create an assistant](/reference/api/2025-01/assistant/create_assistant) in the `eu` region.\n",
      "</Update>  \n",
      "<Update label=\"2024-12-17\" tags={[\"Database\"]}>\n",
      "Document metadata:  {'release': '2024 releases', 'month_year': 'December 2024', 'feature': 'Pinecone Assistant JSON mode and EU region deployment', 'source': 'https://docs.pinecone.io/release-notes/2024.md', 'chunk_num': 2}\n"
     ]
    }
   ],
   "source": [
    "print(\"Document content: \", pinecone_docs[2].page_content)\n",
    "print(\"Document metadata: \", pinecone_docs[2].metadata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document content:  Released [`v2.2.0`](https://github.com/pinecone-io/go-pinecone/releases/tag/v2.2.0) of the [Pinecone Go SDK](/reference/go-sdk). This version adds support for [index tags](/guides/manage-data/manage-indexes#configure-index-tags) when creating or configuring indexes.\n",
      "</Update>\n",
      "Document metadata:  {'release': '2025 releases', 'month_year': 'January 2025', 'feature': 'Released Go SDK v2.2.0', 'source': 'https://docs.pinecone.io/release-notes/2025.md', 'chunk_num': 61}\n"
     ]
    }
   ],
   "source": [
    "print(\"Document content: \", pinecone_docs[-1].page_content)\n",
    "print(\"Document metadata: \", pinecone_docs[-1].metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up Pinecone\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll setup our API keys. For notebooks in Colab environments, we've included a handy block that helps set a Pinecone API key in your environment. In all other contexts, it's sufficient to save your Pinecone and OpenAI keys in your local environment. \n",
    "\n",
    "Run the next two blocks and enter your Pinecone and OpenAI keys as needed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pinecone API key not found in environment.\n"
     ]
    }
   ],
   "source": [
    "\n",
    "import os\n",
    "from getpass import getpass\n",
    "\n",
    "def get_pinecone_api_key():\n",
    "    \"\"\"\n",
    "    Get Pinecone API key from environment variable or prompt user for input.\n",
    "    Returns the API key as a string.\n",
    "\n",
    "    Only necessary for notebooks. When using Pinecone yourself, \n",
    "    you can use environment variables or the like to set your API key.\n",
    "    \"\"\"\n",
    "    api_key = os.environ.get(\"PINECONE_API_KEY\")\n",
    "    \n",
    "    if api_key is None:\n",
    "        try:\n",
    "            # Try Colab authentication if available\n",
    "            from pinecone_notebooks.colab import Authenticate\n",
    "            Authenticate()\n",
    "            # If successful, key will now be in environment\n",
    "            api_key = os.environ.get(\"PINECONE_API_KEY\")\n",
    "        except ImportError:\n",
    "            # If not in Colab or authentication fails, prompt user for API key\n",
    "            print(\"Pinecone API key not found in environment.\")\n",
    "            api_key = getpass(\"Please enter your Pinecone API key: \")\n",
    "            # Save to environment for future use in session\n",
    "            os.environ[\"PINECONE_API_KEY\"] = api_key\n",
    "    \n",
    "    return api_key\n",
    "\n",
    "PINECONE_API_KEY = get_pinecone_api_key()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup OpenAI API Key\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_openai_api_key():\n",
    "    \"\"\"\n",
    "    Get OpenAI API key from environment variable or prompt user for input.\n",
    "    Returns the API key as a string.\n",
    "    \"\"\"\n",
    "\n",
    "    api_key = os.environ.get(\"OPENAI_API_KEY\")\n",
    "    \n",
    "    if api_key is None:\n",
    "        try:\n",
    "            api_key = getpass(\"Please enter your OpenAI API key: \")\n",
    "            # Save to environment for future use in session\n",
    "            os.environ[\"OPENAI_API_KEY\"] = api_key\n",
    "        except Exception as e:\n",
    "            print(f\"Error getting OpenAI API key: {e}\")\n",
    "            return None\n",
    "    \n",
    "    return api_key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "OPENAI_API_KEY = get_openai_api_key()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up Pinecone Index\n",
    "We'll instantiate a Pinecone client, and create an index with a few key properties:\n",
    "- index_name, which identifies our index\n",
    "- dimension, which corresponds to the OpenAI embedding model vector size we'll use\n",
    "- metric, which corresponds to the way \"closeness\" is evaluated with our vectors\n",
    "- a spec, which determines the kind of index we are setting up. In this case, it's a free tier Pinecone serverless index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pinecone import Pinecone\n",
    "\n",
    "pc = Pinecone(\n",
    "        api_key=PINECONE_API_KEY,\n",
    "        # You can remove this parameterfor your own projects\n",
    "        source_tag=\"pinecone_examples:docs:langchain_retrieval_augmentation\"\n",
    "    )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'dimension': 1536,\n",
       " 'index_fullness': 0.0,\n",
       " 'namespaces': {},\n",
       " 'total_vector_count': 0}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index_name = \"langchain-pinecone-rag\"\n",
    "from pinecone import ServerlessSpec\n",
    "\n",
    "if not pc.has_index(index_name):\n",
    "    pc.create_index(\n",
    "        name=index_name,\n",
    "        # dimension of the vector embeddings produced by OpenAI's text-embedding-3-small\n",
    "        dimension=1536,\n",
    "        metric=\"cosine\",\n",
    "        # parameters for the free tier index\n",
    "        spec=ServerlessSpec(\n",
    "            cloud=\"aws\",\n",
    "            region=\"us-east-1\"\n",
    "        )\n",
    "    )\n",
    "\n",
    "# Initialize index client\n",
    "index = pc.Index(name=index_name)\n",
    "\n",
    "# View index stats\n",
    "index.describe_index_stats()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embedding our documents and upserting into Pinecone"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll setup our OpenAI embedding model and Pinecone vector database within LangChain. To do this, we import the related abstractions from LangChain and pass in our API keys and model names.\n",
    "\n",
    "After that, we generate ids for each document-chunk to better manage them within Pinecone."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'release': '2024 releases',\n",
       "  'month_year': 'December 2024',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 0},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'December 2024',\n",
       "  'feature': 'Increased namespaces limit',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 1},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'December 2024',\n",
       "  'feature': 'Pinecone Assistant JSON mode and EU region deployment',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 2},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'December 2024',\n",
       "  'feature': 'Released Spark-Pinecone connector v1.2.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 3},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'December 2024',\n",
       "  'feature': 'New integration with HoneyHive',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 4},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'December 2024',\n",
       "  'feature': 'Released Python SDK v5.4.2',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 5},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'December 2024',\n",
       "  'feature': 'Launch week: Pinecone Local',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 6},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'December 2024',\n",
       "  'feature': 'Launch week: Enhanced security and access controls',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 7},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'December 2024',\n",
       "  'feature': 'Launch week: `pinecone-rerank-v0` and `cohere-rerank-3.5` on Pinecone Inference',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 8},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'December 2024',\n",
       "  'feature': 'Launch week: Integrated Inference',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 9},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'December 2024',\n",
       "  'feature': 'Released .NET SDK v2.1.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 10},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'December 2024',\n",
       "  'feature': 'Improved batch deletion guidance',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 11},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'December 2024',\n",
       "  'feature': 'Launch week: Released `pinecone-sparse-english-v0`',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 12},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'November 2024',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 13},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'November 2024',\n",
       "  'feature': 'Pinecone docs: New workflows and best practices',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 14},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'November 2024',\n",
       "  'feature': 'Released Java SDK v3.1.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 15},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'November 2024',\n",
       "  'feature': 'Pinecone Assistant: Context snippets and structured data files',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 16},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'November 2024',\n",
       "  'feature': 'Monthly spend alerts',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 17},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'November 2024',\n",
       "  'feature': 'Released .NET SDK v2.0.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 18},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'November 2024',\n",
       "  'feature': 'Released Python SDK v5.4.0 and v5.4.1',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 19},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'November 2024',\n",
       "  'feature': 'Enabled CSV export of usage and costs',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 20},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'November 2024',\n",
       "  'feature': 'Added Support chat in the console',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 21},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'November 2024',\n",
       "  'feature': 'Published Assistant quickstart guide',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 22},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'October 2024',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 23},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'October 2024',\n",
       "  'feature': 'Cequence released updated Scala SDK',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 24},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'October 2024',\n",
       "  'feature': 'Added index tagging for categorization',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 25},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'October 2024',\n",
       "  'feature': 'Released major SDK updates: Node.js, Go, Java, and Python',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 26},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'October 2024',\n",
       "  'feature': 'Pinecone API version `2024-10` is now the latest stable version',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 27},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'October 2024',\n",
       "  'feature': 'Pinecone Inference now available on the free Starter plan',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 28},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'October 2024',\n",
       "  'feature': 'Customer-managed encryption keys (CMEK) in early access',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 29},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'October 2024',\n",
       "  'feature': 'Serverless index monitoring generally available',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 30},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'October 2024',\n",
       "  'feature': 'Data import from Amazon S3 in public preview',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 31},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'October 2024',\n",
       "  'feature': 'Chat and update features added to Assistant',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 32},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'September 2024',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 33},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'August 2024',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 34},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'July 2024',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 35},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'June 2024',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 36},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'May 2024',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 37},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'April 2024',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 38},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'March 2024',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 39},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'February 2024',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 40},\n",
       " {'release': '2024 releases',\n",
       "  'month_year': 'January 2024',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2024.md',\n",
       "  'chunk_num': 41},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'August 2025',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 0},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'August 2025',\n",
       "  'feature': 'Data import from Azure Blob Storage',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 1},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'August 2025',\n",
       "  'feature': 'Assistant MCP server endpoint update',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 2},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'August 2025',\n",
       "  'feature': 'VoltAgent integration',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 3},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'July 2025',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 4},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'July 2025',\n",
       "  'feature': 'Increased context window for `pinecone-sparse-english-v0`',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 5},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'July 2025',\n",
       "  'feature': 'Release Go SDK v4.1.0, v4.1.1, and v4.1.2',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 6},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'July 2025',\n",
       "  'feature': 'Release Node.js SDK v6.1.2',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 7},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'June 2025',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 8},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'June 2025',\n",
       "  'feature': 'Unlimited assistant file storage for paid plans',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 9},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'June 2025',\n",
       "  'feature': 'Data import from Google Cloud Storage',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 10},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'June 2025',\n",
       "  'feature': 'Released Python SDK v7.1.0, v7.2.0, and v7.3.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 11},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'June 2025',\n",
       "  'feature': 'Released Go SDK v4.0.0 and v4.0.1',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 12},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'June 2025',\n",
       "  'feature': 'Released Node.js SDK v6.1.1',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 13},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'June 2025',\n",
       "  'feature': 'Data modeling guide',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 14},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'June 2025',\n",
       "  'feature': 'Released Java SDK v5.1.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 15},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'June 2025',\n",
       "  'feature': 'Released Node.js SDK v6.1.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 16},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'May 2025',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 17},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'May 2025',\n",
       "  'feature': 'Released Python SDK v7.0.1 and v7.0.2',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 18},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'May 2025',\n",
       "  'feature': 'Released Node.s SDK v6.0.1',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 19},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'May 2025',\n",
       "  'feature': 'Pinecone API version `2025-04` is now the latest stable version',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 20},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'May 2025',\n",
       "  'feature': 'Released Python SDK v7.0.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 21},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'May 2025',\n",
       "  'feature': 'Released Node.js SDK v6.0.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 22},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'May 2025',\n",
       "  'feature': 'Released Java SDK v5.0.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 23},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'May 2025',\n",
       "  'feature': 'Released .NET SDK v4.0.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 24},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'May 2025',\n",
       "  'feature': 'New Docs IA',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 25},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'April 2025',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 26},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'April 2025',\n",
       "  'feature': 'Bring Your Own Cloud (BYOC) in GCP',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 27},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'April 2025',\n",
       "  'feature': 'Integrate AI agents with Pinecone MCP',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 28},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'April 2025',\n",
       "  'feature': 'Add context to AI agents with Assistant MCP',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 29},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'April 2025',\n",
       "  'feature': 'Upload a file from an in-memory binary stream',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 30},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'April 2025',\n",
       "  'feature': 'Released Pinecone Terraform Provider v1.0.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 31},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'April 2025',\n",
       "  'feature': 'Released .NET SDK v3.1.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 32},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'April 2025',\n",
       "  'feature': 'LLM shortcuts for Pinecone docs',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 33},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'March 2025',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 34},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'March 2025',\n",
       "  'feature': 'Control the context snippets the assistant sends to the LLM',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 35},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'March 2025',\n",
       "  'feature': 'Released Go SDK v3.1.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 36},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'March 2025',\n",
       "  'feature': 'Launch week: Dark mode',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 37},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'March 2025',\n",
       "  'feature': 'Launch week: Self-service audit logs',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 38},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'March 2025',\n",
       "  'feature': 'Launch week: Introducing the Admin API and service accounts',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 39},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'March 2025',\n",
       "  'feature': 'Launch week: Back up an index through the API',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 40},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'March 2025',\n",
       "  'feature': 'Launch week: Optimized database architecture',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 41},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'March 2025',\n",
       "  'feature': 'Firebase Genkit integration',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 42},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'March 2025',\n",
       "  'feature': 'Bring Your Own Cloud (BYOC) in public preview',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 43},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'February 2025',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 44},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'February 2025',\n",
       "  'feature': 'Docs site refresh',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 45},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'February 2025',\n",
       "  'feature': 'Limit the number of chunks retrieved',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 46},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'February 2025',\n",
       "  'feature': 'Assistant Quickstart colab notebook',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 47},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'February 2025',\n",
       "  'feature': 'Released Node.js SDK v5.0.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 48},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'February 2025',\n",
       "  'feature': 'New integrations',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 49},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'February 2025',\n",
       "  'feature': 'Citation highlights in assistant responses',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 50},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'February 2025',\n",
       "  'feature': 'Pinecone API version `2025-01` is now the latest stable version',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 51},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'February 2025',\n",
       "  'feature': 'Released Python SDK v6.0.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 52},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'February 2025',\n",
       "  'feature': 'Released Java SDK v4.0.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 53},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'February 2025',\n",
       "  'feature': 'Released Go SDK v3.0.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 54},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'February 2025',\n",
       "  'feature': 'Released .NET SDK v3.0.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 55},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'January 2025',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 56},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'January 2025',\n",
       "  'feature': 'Update to the API keys page',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 57},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'January 2025',\n",
       "  'feature': 'Sparse-only indexes in early access',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 58},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'January 2025',\n",
       "  'feature': 'Released Node SDK v4.1.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 59},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'January 2025',\n",
       "  'feature': 'New Billing Admin user role',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 60},\n",
       " {'release': '2025 releases',\n",
       "  'month_year': 'January 2025',\n",
       "  'feature': 'Released Go SDK v2.2.0',\n",
       "  'source': 'https://docs.pinecone.io/release-notes/2025.md',\n",
       "  'chunk_num': 61}]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[doc.metadata for doc in pinecone_docs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['release_2024#feature_na#chunk_num0',\n",
       " 'release_2024#feature_Increased namespaces limit#chunk_num1',\n",
       " 'release_2024#feature_Pinecone Assistant JSON mode and EU region deployment#chunk_num2',\n",
       " 'release_2024#feature_Released Spark-Pinecone connector v1.2.0#chunk_num3',\n",
       " 'release_2024#feature_New integration with HoneyHive#chunk_num4',\n",
       " 'release_2024#feature_Released Python SDK v5.4.2#chunk_num5',\n",
       " 'release_2024#feature_Launch week: Pinecone Local#chunk_num6',\n",
       " 'release_2024#feature_Launch week: Enhanced security and access controls#chunk_num7',\n",
       " 'release_2024#feature_Launch week: `pinecone-rerank-v0` and `cohere-rerank-3.5` on Pinecone Inference#chunk_num8',\n",
       " 'release_2024#feature_Launch week: Integrated Inference#chunk_num9',\n",
       " 'release_2024#feature_Released .NET SDK v2.1.0#chunk_num10',\n",
       " 'release_2024#feature_Improved batch deletion guidance#chunk_num11',\n",
       " 'release_2024#feature_Launch week: Released `pinecone-sparse-english-v0`#chunk_num12',\n",
       " 'release_2024#feature_na#chunk_num13',\n",
       " 'release_2024#feature_Pinecone docs: New workflows and best practices#chunk_num14',\n",
       " 'release_2024#feature_Released Java SDK v3.1.0#chunk_num15',\n",
       " 'release_2024#feature_Pinecone Assistant: Context snippets and structured data files#chunk_num16',\n",
       " 'release_2024#feature_Monthly spend alerts#chunk_num17',\n",
       " 'release_2024#feature_Released .NET SDK v2.0.0#chunk_num18',\n",
       " 'release_2024#feature_Released Python SDK v5.4.0 and v5.4.1#chunk_num19',\n",
       " 'release_2024#feature_Enabled CSV export of usage and costs#chunk_num20',\n",
       " 'release_2024#feature_Added Support chat in the console#chunk_num21',\n",
       " 'release_2024#feature_Published Assistant quickstart guide#chunk_num22',\n",
       " 'release_2024#feature_na#chunk_num23',\n",
       " 'release_2024#feature_Cequence released updated Scala SDK#chunk_num24',\n",
       " 'release_2024#feature_Added index tagging for categorization#chunk_num25',\n",
       " 'release_2024#feature_Released major SDK updates: Node.js, Go, Java, and Python#chunk_num26',\n",
       " 'release_2024#feature_Pinecone API version `2024-10` is now the latest stable version#chunk_num27',\n",
       " 'release_2024#feature_Pinecone Inference now available on the free Starter plan#chunk_num28',\n",
       " 'release_2024#feature_Customer-managed encryption keys (CMEK) in early access#chunk_num29',\n",
       " 'release_2024#feature_Serverless index monitoring generally available#chunk_num30',\n",
       " 'release_2024#feature_Data import from Amazon S3 in public preview#chunk_num31',\n",
       " 'release_2024#feature_Chat and update features added to Assistant#chunk_num32',\n",
       " 'release_2024#feature_na#chunk_num33',\n",
       " 'release_2024#feature_na#chunk_num34',\n",
       " 'release_2024#feature_na#chunk_num35',\n",
       " 'release_2024#feature_na#chunk_num36',\n",
       " 'release_2024#feature_na#chunk_num37',\n",
       " 'release_2024#feature_na#chunk_num38',\n",
       " 'release_2024#feature_na#chunk_num39',\n",
       " 'release_2024#feature_na#chunk_num40',\n",
       " 'release_2024#feature_na#chunk_num41',\n",
       " 'release_2025#feature_na#chunk_num0',\n",
       " 'release_2025#feature_Data import from Azure Blob Storage#chunk_num1',\n",
       " 'release_2025#feature_Assistant MCP server endpoint update#chunk_num2',\n",
       " 'release_2025#feature_VoltAgent integration#chunk_num3',\n",
       " 'release_2025#feature_na#chunk_num4',\n",
       " 'release_2025#feature_Increased context window for `pinecone-sparse-english-v0`#chunk_num5',\n",
       " 'release_2025#feature_Release Go SDK v4.1.0, v4.1.1, and v4.1.2#chunk_num6',\n",
       " 'release_2025#feature_Release Node.js SDK v6.1.2#chunk_num7',\n",
       " 'release_2025#feature_na#chunk_num8',\n",
       " 'release_2025#feature_Unlimited assistant file storage for paid plans#chunk_num9',\n",
       " 'release_2025#feature_Data import from Google Cloud Storage#chunk_num10',\n",
       " 'release_2025#feature_Released Python SDK v7.1.0, v7.2.0, and v7.3.0#chunk_num11',\n",
       " 'release_2025#feature_Released Go SDK v4.0.0 and v4.0.1#chunk_num12',\n",
       " 'release_2025#feature_Released Node.js SDK v6.1.1#chunk_num13',\n",
       " 'release_2025#feature_Data modeling guide#chunk_num14',\n",
       " 'release_2025#feature_Released Java SDK v5.1.0#chunk_num15',\n",
       " 'release_2025#feature_Released Node.js SDK v6.1.0#chunk_num16',\n",
       " 'release_2025#feature_na#chunk_num17',\n",
       " 'release_2025#feature_Released Python SDK v7.0.1 and v7.0.2#chunk_num18',\n",
       " 'release_2025#feature_Released Node.s SDK v6.0.1#chunk_num19',\n",
       " 'release_2025#feature_Pinecone API version `2025-04` is now the latest stable version#chunk_num20',\n",
       " 'release_2025#feature_Released Python SDK v7.0.0#chunk_num21',\n",
       " 'release_2025#feature_Released Node.js SDK v6.0.0#chunk_num22',\n",
       " 'release_2025#feature_Released Java SDK v5.0.0#chunk_num23',\n",
       " 'release_2025#feature_Released .NET SDK v4.0.0#chunk_num24',\n",
       " 'release_2025#feature_New Docs IA#chunk_num25',\n",
       " 'release_2025#feature_na#chunk_num26',\n",
       " 'release_2025#feature_Bring Your Own Cloud (BYOC) in GCP#chunk_num27',\n",
       " 'release_2025#feature_Integrate AI agents with Pinecone MCP#chunk_num28',\n",
       " 'release_2025#feature_Add context to AI agents with Assistant MCP#chunk_num29',\n",
       " 'release_2025#feature_Upload a file from an in-memory binary stream#chunk_num30',\n",
       " 'release_2025#feature_Released Pinecone Terraform Provider v1.0.0#chunk_num31',\n",
       " 'release_2025#feature_Released .NET SDK v3.1.0#chunk_num32',\n",
       " 'release_2025#feature_LLM shortcuts for Pinecone docs#chunk_num33',\n",
       " 'release_2025#feature_na#chunk_num34',\n",
       " 'release_2025#feature_Control the context snippets the assistant sends to the LLM#chunk_num35',\n",
       " 'release_2025#feature_Released Go SDK v3.1.0#chunk_num36',\n",
       " 'release_2025#feature_Launch week: Dark mode#chunk_num37',\n",
       " 'release_2025#feature_Launch week: Self-service audit logs#chunk_num38',\n",
       " 'release_2025#feature_Launch week: Introducing the Admin API and service accounts#chunk_num39',\n",
       " 'release_2025#feature_Launch week: Back up an index through the API#chunk_num40',\n",
       " 'release_2025#feature_Launch week: Optimized database architecture#chunk_num41',\n",
       " 'release_2025#feature_Firebase Genkit integration#chunk_num42',\n",
       " 'release_2025#feature_Bring Your Own Cloud (BYOC) in public preview#chunk_num43',\n",
       " 'release_2025#feature_na#chunk_num44',\n",
       " 'release_2025#feature_Docs site refresh#chunk_num45',\n",
       " 'release_2025#feature_Limit the number of chunks retrieved#chunk_num46',\n",
       " 'release_2025#feature_Assistant Quickstart colab notebook#chunk_num47',\n",
       " 'release_2025#feature_Released Node.js SDK v5.0.0#chunk_num48',\n",
       " 'release_2025#feature_New integrations#chunk_num49',\n",
       " 'release_2025#feature_Citation highlights in assistant responses#chunk_num50',\n",
       " 'release_2025#feature_Pinecone API version `2025-01` is now the latest stable version#chunk_num51',\n",
       " 'release_2025#feature_Released Python SDK v6.0.0#chunk_num52',\n",
       " 'release_2025#feature_Released Java SDK v4.0.0#chunk_num53',\n",
       " 'release_2025#feature_Released Go SDK v3.0.0#chunk_num54',\n",
       " 'release_2025#feature_Released .NET SDK v3.0.0#chunk_num55',\n",
       " 'release_2025#feature_na#chunk_num56',\n",
       " 'release_2025#feature_Update to the API keys page#chunk_num57',\n",
       " 'release_2025#feature_Sparse-only indexes in early access#chunk_num58',\n",
       " 'release_2025#feature_Released Node SDK v4.1.0#chunk_num59',\n",
       " 'release_2025#feature_New Billing Admin user role#chunk_num60',\n",
       " 'release_2025#feature_Released Go SDK v2.2.0#chunk_num61']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "from langchain_pinecone import PineconeVectorStore\n",
    "\n",
    "embeddings = OpenAIEmbeddings(api_key=OPENAI_API_KEY, model=\"text-embedding-3-small\")\n",
    "\n",
    "vector_store = PineconeVectorStore(index=index, embedding=embeddings)\n",
    "\n",
    "\n",
    "# do url_title, chunk_num to enable subscriptable hashing and replacement\n",
    "\n",
    "def clean_url_for_title(url):\n",
    "    # grabs the end of the url minus .md\n",
    "    return url.split(\"/\")[-1].replace(\".md\", \"\")\n",
    "\n",
    "# Here, we follow a schema that puts the document name, and the chunk number together, like doc1#chunk1\n",
    "\n",
    "def generate_ids(doc_chunk):\n",
    "\n",
    "    title = clean_url_for_title(doc_chunk.metadata['source'])\n",
    "    chunk_num = doc_chunk.metadata['chunk_num']\n",
    "    feature = doc_chunk.metadata['feature'] if 'feature' in doc_chunk.metadata else \"na\"\n",
    "    return f\"release_{title}#feature_{feature}#chunk_num{chunk_num}\"\n",
    "\n",
    "ids = [generate_ids(doc) for doc in pinecone_docs]\n",
    "\n",
    "\n",
    "# To learn more, look here: https://docs.pinecone.io/guides/index-data/data-modeling\n",
    "\n",
    "vector_store.add_documents(documents=pinecone_docs, ids=ids)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A5dDWPGoIrd9"
   },
   "source": [
    "## Bringing it all together: Using OpenAI to learn about our releases\n",
    "\n",
    "Finally, we'll setup an OpenAI LLM endpoint to generate responses given a user's query about our release notes!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chat_models import init_chat_model\n",
    "\n",
    "\n",
    "# Note that as of 08-12-2025, the cutoff for GPT-5 is Sept 29, 2024.\n",
    "# If you don't have access yet, gpt-4o-mini works great!\n",
    "\n",
    "llm = init_chat_model(\"gpt-5\", model_provider=\"openai\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's run a query and retrieve some documents. These will be what is ultimately passed to our LLM that uses Pinecone to answer queries. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# OpenAI models will be unable to answer this due to training cutoffs in 2023-2024\n",
    "\n",
    "query = \"Tell me about version 7.0 of the Pinecone Python SDK\"\n",
    "\n",
    "retrieved_docs = vector_store.similarity_search(query, k=5)\n",
    "docs_content = \"\\n\\n\".join(doc.page_content for doc in retrieved_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sjOBIQ5rFeLZ"
   },
   "source": [
    "We can peek at the retrieved documents to confirm up to date information is being passed in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc number: 1\n",
      "Released [`v7.0.1`](https://github.com/pinecone-io/pinecone-python-client/releases/tag/v7.0.1) and [`v7.0.2`](https://github.com/pinecone-io/pinecone-python-client/releases/tag/v7.0.2) of the [Pinecone Python SDK](/reference/python-sdk). These versions fix minor bugs discovered since the release of the `v7.0.0` major version.\n",
      "</Update>  \n",
      "<Update label=\"2025-05-29\" tags={[\"SDK\"]}>\n",
      "Metadata:\n",
      "{'chunk_num': 18.0, 'feature': 'Released Python SDK v7.0.1 and v7.0.2', 'month_year': 'May 2025', 'release': '2025 releases', 'source': 'https://docs.pinecone.io/release-notes/2025.md'}\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Doc number: 2\n",
      "Released [`v7.1.0`](https://github.com/pinecone-io/pinecone-python-client/releases/tag/v7.1.0), [`v7.2.0`](https://github.com/pinecone-io/pinecone-python-client/releases/tag/v7.2.0), and [`v7.3.0`](https://github.com/pinecone-io/pinecone-python-client/releases/tag/v7.3.0) of the [Pinecone Python SDK](/reference/python-sdk).  \n",
      "* `v7.1.0` fixes minor bugs.\n",
      "* `v7.2.0` adds support for [managing namespaces](/guides/manage-data/manage-namespaces).\n",
      "* `v7.3.0` adds support for admin API operations for working with API keys, projects, and service accounts.\n",
      "</Update>  \n",
      "<Update label=\"2025-06-16\" tags={[\"SDK\"]}>\n",
      "Metadata:\n",
      "{'chunk_num': 11.0, 'feature': 'Released Python SDK v7.1.0, v7.2.0, and v7.3.0', 'month_year': 'June 2025', 'release': '2025 releases', 'source': 'https://docs.pinecone.io/release-notes/2025.md'}\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Doc number: 3\n",
      "Released [`v7.0.0`](https://github.com/pinecone-io/pinecone-python-client/releases/tag/v7.0.0) of the [Pinecone Python SDK](/reference/python-sdk). This version uses the latest stable API version, `2025-04`, and includes support for the following:  \n",
      "* [Creating and managing backups](/guides/manage-data/back-up-an-index)\n",
      "* [Restoring indexes from backups](/guides/manage-data/restore-an-index)\n",
      "* [Listing embedding and reranking models hosted by Pinecone](/reference/api/2025-04/inference/list_models)\n",
      "* [Getting details about a model hosted by Pinecone](/reference/api/2025-04/inference/describe_model)\n",
      "* [Creating a BYOC index](/guides/production/bring-your-own-cloud#create-an-index)  \n",
      "Additionally, the `pinecone-plugin-assistant` package required to work with [Pinecone Assistant](/guides/assistant/overview) is now included by default; it is no longer necessary to install the plugin separately.\n",
      "</Update>  \n",
      "<Update label=\"2025-05-19\" tags={[\"SDK\"]}>\n",
      "Metadata:\n",
      "{'chunk_num': 21.0, 'feature': 'Released Python SDK v7.0.0', 'month_year': 'May 2025', 'release': '2025 releases', 'source': 'https://docs.pinecone.io/release-notes/2025.md'}\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Doc number: 4\n",
      "Released [`v6.0.0`](https://github.com/pinecone-io/pinecone-python-client/releases/tag/v6.0.0) of the [Pinecone Python SDK](/reference/python-sdk). This version uses the latest stable API version, `2025-01`, and includes support for the following:  \n",
      "* [Index tags](/guides/manage-data/manage-indexes#configure-index-tags) to categorize and identify your indexes.\n",
      "* [Integrated inference](/reference/api/introduction#inference) without the need for extra plugins. If you were using the preview functionality of integrated inference, you must uninstall the `pinecone-plugin-records` package to use the `v6.0.0` release.\n",
      "* Enum objects to help with the discoverability of some configuration options, for example, `Metric`, `AwsRegion`, `GcpRegion`, `PodType`, `EmbedModel`, `RerankModel`. This is a backwards compatible change; you can still pass string values for affected fields.\n",
      "* New client variants, `PineconeAsyncio` and `IndexAsyncio`, which provide `async` methods for use with [asyncio](https://docs.python.org/3/library/asyncio.html). This makes it possible to use Pinecone with modern async web frameworks such as [FastAPI](https://fastapi.tiangolo.com/), [Quart](https://quart.palletsprojects.com/en/latest/), and [Sanic](https://sanic.dev/en/). Async support should significantly increase the efficiency of running many upserts in parallel.  \n",
      "<Warning>\n",
      "Before upgrading to `v6.0.0`, update all relevant code to account for the following [breaking changes](/reference/api/versioning#breaking-changes). See the [`v6.0.0`](https://github.com/pinecone-io/pinecone-python-client/releases/tag/v6.0.0) release notes for full details.  \n",
      "* Incorporated the `pinecone-plugin-records` and `pinecone-plugin-inference` plugins into the `pinecone` package. If you are using these plugins, you must unstall them to use `v6.0.0`.\n",
      "* Dropped support for Python 3.8, which has now reached official end of life, and added support for Python 3.13.\n",
      "* Removed the explicit dependency on `tqdm`, which is used to provide a progress bar when upserting data into Pinecone. If `tqdm` is available in the environment, the Pinecone SDK will detect and use it, but `tdqm` is no longer required to run the SDK. Popular notebook platforms such as [Jupyter](https://jupyter.org/) and [Google Colab](https://colab.google/) already include `tqdm` in the environment by default, but if you are running small scripts in other environments and want to continue seeing progress bars, you will need to separately install the `tqdm` package.\n",
      "* Removed some previously deprecated and rarely used keyword arguments (`config`, `openapi_config`, and `index_api`) to instead prefer dedicated keyword arguments for individual settings such as `api_key`, `proxy_url`, etc.\n",
      "</Warning>\n",
      "</Update>  \n",
      "<Update label=\"2025-02-07\" tags={[\"SDK\"]}>\n",
      "Metadata:\n",
      "{'chunk_num': 52.0, 'feature': 'Released Python SDK v6.0.0', 'month_year': 'February 2025', 'release': '2025 releases', 'source': 'https://docs.pinecone.io/release-notes/2025.md'}\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Doc number: 5\n",
      "Released [`v6.0.0`](https://github.com/pinecone-io/pinecone-ts-client/releases/tag/v6.0.0) of the [Pinecone Node.js SDK](/reference/node-sdk). This version uses the latest stable API version, `2025-04`, and includes support for the following:  \n",
      "* [Managing namespaces](/guides/manage-data/manage-namespaces)\n",
      "* [Creating and managing backups](/guides/manage-data/back-up-an-index)\n",
      "* [Restoring indexes from backups](/guides/manage-data/restore-an-index)\n",
      "* [Listing embedding and reranking models hosted by Pinecone](/reference/api/2025-04/inference/list_models)\n",
      "* [Getting details about a model hosted by Pinecone](/reference/api/2025-04/inference/describe_model)\n",
      "</Update>  \n",
      "<Update label=\"2025-05-19\" tags={[\"SDK\"]}>\n",
      "Metadata:\n",
      "{'chunk_num': 22.0, 'feature': 'Released Node.js SDK v6.0.0', 'month_year': 'May 2025', 'release': '2025 releases', 'source': 'https://docs.pinecone.io/release-notes/2025.md'}\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "for num, d in enumerate(retrieved_docs):\n",
    "    print(f\"Doc number: {num+1}\")\n",
    "    print(d.page_content)\n",
    "    print(\"Metadata:\")\n",
    "    print(d.metadata)\n",
    "    print(\"-\"*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing responses with/without RAG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Without our RAG pipeline with release notes indexed, OpenAI models will be\n",
    "unable to answer questions about new versions of Pinecone. They may even \"hallucinate\", or fabricate information about the versions that may not exist!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I don’t have release notes for Pinecone’s Python SDK version 7.0 in my training data (my knowledge goes up to Oct 2024), and I don’t want to guess. A couple of quick clarifications and ways to get the exact details:\n",
      "\n",
      "Questions\n",
      "- Do you mean the “pinecone” package or “pinecone-client”? They are different packages, and their version numbers may not match.\n",
      "- Are you looking for new features, breaking changes, or migration guidance?\n",
      "\n",
      "How to find the v7.0 details right now\n",
      "- PyPI release notes:\n",
      "  - pinecone: https://pypi.org/project/pinecone/\n",
      "  - pinecone-client: https://pypi.org/project/pinecone-client/\n",
      "- GitHub changelog (pinecone-io org): look for the repository of the package you use (commonly “pinecone-python”) and open CHANGELOG.md or Releases.\n",
      "- Docs: https://docs.pinecone.io (search “Python SDK” and “migration guide”).\n",
      "\n",
      "Commands to check and upgrade locally\n",
      "- Check which package/version you have:\n",
      "  - pip show pinecone\n",
      "  - pip show pinecone-client\n",
      "  - python -c \"import pinecone, sys; print('pinecone', getattr(pinecone, '__version__', 'unknown'))\"\n",
      "- Upgrade:\n",
      "  - pip install -U pinecone\n",
      "  - or pip install -U pinecone-client\n",
      "\n",
      "What to look for in a major (7.0) release\n",
      "- Breaking API changes (method/class renames or moved modules)\n",
      "- Deprecations/removals (older index types or parameters)\n",
      "- Auth/init changes (e.g., Pinecone(api_key=...) vs init functions)\n",
      "- Index create specs and defaults (dimension/metric, serverless vs pod settings)\n",
      "- Data model tweaks (upsert/query payload/response shapes)\n",
      "- Retry/timeout behavior and error classes\n",
      "- New features (batching, streaming, filters, hybrid/search, inference hooks)\n",
      "\n",
      "If you can share:\n",
      "- Which package (“pinecone” or “pinecone-client”)\n",
      "- The current version you’re on\n",
      "- A snippet of how you currently initialize, upsert, and query\n",
      "\n",
      "…I can map that to 7.0 and give you precise, step-by-step migration guidance.\n"
     ]
    }
   ],
   "source": [
    "print(llm.invoke(query).content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, with our pipeline in place, we get an answer that is more likely to be correct, and definitely grounded. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here’s what’s in the Pinecone Python SDK 7.0 line:\n",
      "\n",
      "- v7.0.0 (2025-06-16): Moves to API version 2025-04 and adds:\n",
      "  - Creating and managing backups\n",
      "  - Restoring indexes from backups\n",
      "  - Listing and describing Pinecone-hosted embedding and reranking models\n",
      "  - Creating Bring Your Own Cloud (BYOC) indexes\n",
      "  - pinecone-plugin-assistant included by default (no separate install needed)\n",
      "- v7.0.1 and v7.0.2: Patch releases that fix minor bugs discovered after v7.0.0.\n"
     ]
    }
   ],
   "source": [
    "prompt = f'''You are an assistant that answers questions exclusively about the \n",
    "Pinecone SDK release notes:\n",
    "\n",
    "Here's a question: {query}\n",
    "\n",
    "Here's some context from the release notes:\n",
    "\n",
    "{docs_content}\n",
    "\n",
    "Answer:\n",
    "'''\n",
    "\n",
    "# This will take a few seconds to run, due to the generation of the response from OpenAI\n",
    "answer = llm.invoke(prompt)\n",
    "\n",
    "print(answer.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapping it all in a function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_response(query, use_pinecone=False):\n",
    "    # Function to easily generate a response with and without Pinecone data\n",
    "\n",
    "    if use_pinecone:\n",
    "        retrieved_docs = vector_store.similarity_search(query, k=5)\n",
    "        docs_content = \"\\n\\n\".join(doc.page_content for doc in retrieved_docs)\n",
    "        prompt = f'''\n",
    "        You are an assistant that answers questions exclusively about the \n",
    "        Pinecone SDK release notes:\n",
    "\n",
    "        Here's a question: {query}\n",
    "\n",
    "        Here's some context from the release notes:\n",
    "\n",
    "        {docs_content}\n",
    "\n",
    "        Answer: '''\n",
    "\n",
    "        #print out retrieved documents\n",
    "        print(\"Retrieved documents:.....\")\n",
    "        for num, d in enumerate(retrieved_docs):\n",
    "            print(f\"Doc number: {num+1}\")\n",
    "            print(d.page_content)\n",
    "            print(\"Metadata:\")\n",
    "            print(d.metadata)\n",
    "            print(\"-\"*100)\n",
    "        print(\"Chatbot response:.....\")\n",
    "\n",
    "        return llm.invoke(prompt).content\n",
    "    else:\n",
    "        # no context is passed\n",
    "        return llm.invoke(query).content\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Short answer: Pinecone’s sparse encoder used to be effectively limited to ~512 tokens (BERT-era limit). Recently, Pinecone raised the usable context window by adding server-side chunk-and-merge, so you can pass much longer inputs and get a single sparse vector back. In practice, this means you no longer need to manually chunk long documents just to get a usable sparse embedding.\n",
      "\n",
      "What changed\n",
      "- Longer inputs accepted: You can now send texts far beyond 512 tokens; the service splits them internally and merges the term weights into one sparse vector. This improves recall on long documents and simplifies pipelines.\n",
      "- Smarter merging: The merger reduces double-counting of repeated terms across chunks and normalizes weights, helping stability vs. naive summation.\n",
      "- Progressive limits: The maximum accepted input length has been increased from the original 512-token ceiling to a much larger cap (the exact hard limit can change as they roll out updates).\n",
      "\n",
      "How to confirm the current limit\n",
      "- Check the pinecone-sparse model card in Pinecone’s docs or the Inference “Models” page.\n",
      "- Call the models/list or embeddings create endpoint and look for the model’s max input length; if you exceed it, the API error usually reports the current cap.\n",
      "- Review the Pinecone changelog or release notes for “pinecone-sparse” (they’ve been incrementally expanding the window).\n",
      "\n",
      "Practical guidance\n",
      "- You can now pass full sections/long pages directly. If you still prefer control, client-side chunking at 800–1,200 tokens with slight overlap works well; the server-side merge will still help.\n",
      "- For hybrid search, keep an eye on term-weight saturation with very long inputs; consider light client-side chunking if you need fine-grained retrieval.\n",
      "\n",
      "If you can share the timeframe or a link to the announcement you’re referring to, I can tailor this to the exact version and hard token limit you’re targeting.\n"
     ]
    }
   ],
   "source": [
    "query = \"Tell me about recent changes to the pinecone-sparse embedding model context window\"\n",
    "\n",
    "print(generate_response(query, use_pinecone=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Retrieved documents:.....\n",
      "Doc number: 1\n",
      "You can now raise the context window for Pinecone's hosted [`pinecone-sparse-english-v0`](/guides/index-data/create-an-index#pinecone-sparse-english-v0) embedding model from `512` to `2048` using the `max_tokens_per_sequence` parameter.\n",
      "</Update>  \n",
      "<Update label=\"2025-07-23\" tags={[\"SDK\"]}>\n",
      "Metadata:\n",
      "{'chunk_num': 5.0, 'feature': 'Increased context window for `pinecone-sparse-english-v0`', 'month_year': 'July 2025', 'release': '2025 releases', 'source': 'https://docs.pinecone.io/release-notes/2025.md'}\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Doc number: 2\n",
      "Pinecone Inference now supports [`pinecone-sparse-english-v0`](/guides/search/rerank-results#pinecone-sparse-english-v0), Pinecone's sparse embedding model, which estimates the lexical importance of tokens by leveraging their context, unlike traditional retrieval models like BM25, which rely solely on term frequency. This model is in [public preview](/release-notes/feature-availability).\n",
      "</Update>\n",
      "Metadata:\n",
      "{'chunk_num': 12.0, 'feature': 'Launch week: Released `pinecone-sparse-english-v0`', 'month_year': 'December 2024', 'release': '2024 releases', 'source': 'https://docs.pinecone.io/release-notes/2024.md'}\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Doc number: 3\n",
      "Released [`v3.0.0`](https://github.com/pinecone-io/pinecone-dotnet-client/releases/tag/3.0.0) of the [Pinecone .NET SDK](/reference/dotnet-sdk). This version uses the latest stable API version, `2025-01`, and adds support for [sparse-only indexes](/guides/index-data/indexing-overview#sparse-indexes).  \n",
      "<Warning>\n",
      "Before upgrading to `v3.0.0`, update all relevant code to account for the following [breaking changes](/reference/api/versioning#breaking-changes). See the [`v3.0.0`](https://github.com/pinecone-io/pinecone-dotnet-client/releases/tag/3.0.0) release notes for full details.  \n",
      "* [`embed` operation](/reference/api/2025-01/inference/generate-embeddings):\n",
      "* The `Embedding` type has changed from a simple object to a discriminated union, supporting both `DenseEmbedding` and `SparseEmbedding`. New helper methods available on the Embedding type: `IsDense` and `IsSparse` for type checking, `AsDense()` and `AsSparse()` for type conversion, and `Match()` and `Visit()` for pattern matching.\n",
      "* The `Parameters` property now uses `Dictionary<string, object?>?` instead of `EmbedRequestParameters`.  \n",
      "* `rerank` operation:\n",
      "* The `Document` property now uses `Dictionary<string, object?>?` instead of `Dictionary<string, string>?`.\n",
      "* The `Parameters` property now uses `Dictionary<string, object?>?` instead of `Dictionary<string, string>?`.\n",
      "</Warning>\n",
      "</Update>\n",
      "Metadata:\n",
      "{'chunk_num': 55.0, 'feature': 'Released .NET SDK v3.0.0', 'month_year': 'February 2025', 'release': '2025 releases', 'source': 'https://docs.pinecone.io/release-notes/2025.md'}\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Doc number: 4\n",
      "Released [`v4.0.0`](https://github.com/pinecone-io/pinecone-java-client/releases/tag/v4.0.0) of the [Pinecone Java SDK](/reference/java-sdk). This version uses the latest stable API version, `2025-01`, and adds support for [sparse-only indexes](/guides/index-data/indexing-overview#sparse-indexes).  \n",
      "<Warning>\n",
      "Before upgrading to `v4.0.0`, update all relevant code to account for the following [breaking changes](/reference/api/versioning#breaking-changes). See the [`v4.0.0`](https://github.com/pinecone-io/pinecone-java-client/releases/tag/v4.0.0) release notes for full details.  \n",
      "* [`embed` method](/reference/api/2025-01/inference/generate-embeddings):\n",
      "* `parameters` now accepts `Map<String, Object>` instead of `EmbedRequestParameters`.\n",
      "* The `Embeddings` response class now has dense and sparse embeddings. You now must use `getDenseEmbedding()` or `getSparseEmbedding()`. For example, instead of `embeddings.getData().get(0).getValues()`, you would use `embeddings.getData().get(0).getDenseEmbedding().getValues()`.  \n",
      "* [`rerank` method](/guides/search/rerank-results):\n",
      "* `documents` now accepts `List<Map<String, Object>>` instead of `List<Map<String, String>>`.\n",
      "* `parameters` now accepts `Map<String, Object>` instead of `Map<String, String>`.\n",
      "</Warning>\n",
      "</Update>  \n",
      "<Update label=\"2025-02-07\" tags={[\"SDK\"]}>\n",
      "Metadata:\n",
      "{'chunk_num': 53.0, 'feature': 'Released Java SDK v4.0.0', 'month_year': 'February 2025', 'release': '2025 releases', 'source': 'https://docs.pinecone.io/release-notes/2025.md'}\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Doc number: 5\n",
      "You can now use [embedding models](/guides/index-data/create-an-index#embedding-models) and [reranking models](/guides/search/rerank-results#reranking-models) hosted on Pinecone as an integrated part of upserting and searching.\n",
      "</Update>  \n",
      "<Update label=\"2024-12-05\" tags={[\"SDK\"]}>\n",
      "Metadata:\n",
      "{'chunk_num': 9.0, 'feature': 'Launch week: Integrated Inference', 'month_year': 'December 2024', 'release': '2024 releases', 'source': 'https://docs.pinecone.io/release-notes/2024.md'}\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Chatbot response:.....\n",
      "- You can now raise the context window for Pinecone’s hosted pinecone-sparse-english-v0 embedding model from 512 to 2048 tokens.\n",
      "- Set the max_tokens_per_sequence parameter to increase the limit.\n"
     ]
    }
   ],
   "source": [
    "print(generate_response(query, use_pinecone=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapping up\n",
    "\n",
    "And that's that! You've successfully implemented retrieval augmented generation with Pinecone, OpenAI and LangChain. Wanna learn more? Try implementing the following:\n",
    "\n",
    "- Sparse search to enable precise time, date and feature recognition in query results\n",
    "- Expanding the set of documents to encompass all Pinecone documentaiotn\n",
    "- Learning how to chunk and process code data, to build your own code assistant\n",
    "\n",
    "To finish, let's delete our index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "id": "PpJp-xExFeLa"
   },
   "outputs": [],
   "source": [
    "pc.delete_index(name=index_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2kNh44bEFeLe"
   },
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "pinecone-examples",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
